In recent years, researches as to mixed reality (hereinafter referred to as MR technique) intended for displaying additional information and virtual objects (hereinafter generically referred to as virtual images) in a superimposed manner in a real space have been vigorously conducted. Among them, attention is being given to systems in which an observer wears a head-mounted display (hereinafter referred to as HMD) of the video see-through type to render virtual images superimposed on real images that are shot by a camera included in or mounted on the HMD with the real space and the virtual space being three-dimensionally registered, and display the resulting mixed reality images (hereinafter referred to as MR images) on the HMD in real time (herein, these systems are referred to as MR systems).
Registration of the virtual image and the real image is a prime challenge in the MR system, and for achieving it, it is necessary to measure accurately the viewpoint position and posture of the camera. Generally, if positions on photographed images at a plurality of points (theoretically three points or more, and six points or more for stable solution) for which three positions are known, the viewpoint position and posture of the camera can be determined from their correspondence relations (Herein, points like these are generically referred to as landmarks). That is, the problem of registration depends on how accurately the landmark is tracked or detected from within the image photographed with a moving camera to obtain its position.
The inventors have previously developed devices applying the MR technique in fields such as games. These devices are based on indoor use.
In indoor uses as described above, characteristic markers (characteristic colors such as red and green arranged in monochrome or in combination, and characteristic patterns such as checked patterns and concentric circles are often used) are arranged in a target space, and are set as landmarks, whereby detection of landmarks by image processing can be performed with ease and stability, and thus accurate registration can be achieved.
As for methods of detecting markers when markers based on colors, for example, a method in which the marker is photographed under a certain illuminating environment, and a representative color of the marker area in the image is extracted and stored, thereby detecting the marker as an area having a color (or its proximate color) same as the representative color of the marker area in the photographed image is known. Also, as for methods of detecting markers when markers based on patterns, for example, each marker is photographed under a certain illuminating environment, and the proximate area of the marker in the image is stored as a template image, whereby the marker can be detected through template matching. That is, similarity is computed between the template image and the partial area of the photographed image to detect the position of the partial area most similar to the template image as the position of the marker. Herein, image characteristics that are used as clues to detect markers such as the representative colors of the marker area and the template image as described above are generically referred to as “detection parameters”.
On the other hand, needs for MR systems based on outdoor uses are also increased including, for example, cases where the virtual image of a guide is displayed on the HMD to give a tour of a college site and a tourist attraction.
In the outdoors, it is often difficult to place a man-made marker in an environment. As for methods of measuring the viewpoint position and posture of the observer under these situations, methods in which points having features capable of being detected through image processing (for example, corners of structures, points with large quantity of texture in the structure, points with hues locally changed) in the photographed image photographed by the camera are used as landmarks are known. For detecting the landmark from the photographed image, a template matching technique can be applied.
However, in the outdoor environment, how the landmark is viewed (brightness and hues) is changed due to changes in environmental light by weather (clear/cloudy/rainy) and time periods (morning/daytime/evening). Thus, there is a disadvantage that when detection of landmarks by template matching is performed, correct matching is not carried out due to changes in environmental light, making it impossible to detect landmarks even if the template image for matching is prepared in advance as the detection parameter. Hence, the problem of being unable to obtain correct viewpoint positions and postures and thus making it impossible to perform correct registration between the real image and the virtual image arises. Also, even when the man-made marker is used in the indoor environment, a similar problem arises in the case where the illuminating environment changes.